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ABSTRACT 


Customer satisfaction is very important for public service providers, customer 
satisfaction can be delivered with a survey application or writing criticism that 
can be used to evaluate and improve service. Unfortunately, there are only a 
few customers who are willing to give an assessment. The survey application 
cannot represent the overall feeling of the customer, so it is necessary to 
analyze the content of the conversation between the customer and the service 
personnel to determine the level of customer satisfaction. In small amounts, it 
can be done manually, but in large quantities it is more effective to use the 
system. A solution is needed in the form of a system that converts voice 
conversations into text and analyzes customer satisfaction to obtain 
information for evaluation and improvement of services. This research uses K- 
nearest neighbors (KNN) and term frequency-inverse document frequency 
(TF-IDF) algorithm with natural language processing (NLP) approach to 
classify conversations into 2 classes, "satisfied" and " dissatisfied ". The results 


of this study received 74.00% accuracy, 76.00% precision and 73.08% recall. 
In conversations with the label "satisfied" shows customers satisfied with the 
service and fulfillment of customer desires, while in conversations with the 
label "not satisfied" customers are less satisfied with the waiting time. 
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1. INTRODUCTION 

Public satisfaction is a determining factor in the quality of public services, so every organization that 
provides public services is expected to be able to provide satisfaction to customers [1]. To find out customer 
satisfaction, the right measuring instrument is needed [2]. Customer satisfaction is usually measured by a 
survey application or critical writing that can be used to evaluate and improve service. Unfortunately, there are 
only a few customers who are willing to give an assessment. Low conditions of public services are 
characterized by public services that are not transparent, discriminatory, complicated, and corrupt. The 
application of public service policy innovations becomes a necessity in realizing quality public services [3]. 

The survey application cannot represent the overall feeling of the customer, it is necessary to analyze 
the contents of the conversation between the customer and the service personnel to determine the level of 
customer satisfaction and absorb and understand customer desires [4]. Another purpose is to prevent 
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discriminatory or corrupt practices in the delivery of services, so that they can deliver services that are fair, 
democratic and friendly [5]. Analysis of service conversations in small amounts can be done manually, but in 
it is more effective to use the system in large quantities. 

Researchers have developed a product called "Kata Kita" which serves to support public services. 
"Kata Kita" has the feature of translating conversations into text and displaying them on the screen to help the 
deaf understand service personnel and sentiment analysis features to assess customer satisfaction from the 
conversation. Sentiment Analysis is used to find out the opinion about a topic which is as positive, negative, 
and neutral sentiments [6]. The purpose of this study is to examine the effectiveness of "Kata Kita" to conduct 
sentiment analysis on public service conversations in one of the hospitals in Yogyakarta, Indonesia. 

Conversations are recorded and converted to text using Speech Recognition System technology. 
Speech recognition system is the process of converting speech signals into word sequences using an algorithm 
implemented as a computer program [7]. The text of the conversation is processed using machine learning to 
be labeled automatically without the need to wait for input from customers, so that the process of evaluating 
and improving services can run effectively. 

Previous research by Lutfi and Permatasari [8] analyzed the marketplace to find out positive or 
negative user sentiment with an approach utilizing support vector machine with an accuracy of 93.42%. In a 
study conducted by Akhmad Deviyanto and Wahyudi [9] presented a sentiment analysis of opinion data on 
Twitter user sentiment analysis on the topic of local elections in the Jakarta of Indonesia in 2017 with an 
accuracy of 67.2%. In another study by Norman Kendal [10] showed the use of natural language processing 
(NLP) can improve the prediction of fashion product categories or sub categories of product titles. 

The purpose of this study is to analyze the sentiments of public service conversation in one of the 
hospitals in Yogyakarta, Indonesia using the uses K-nearest neighbors (KNN) algorithm by weighting term 
frequency-inverse document frequency (TF-IDF), and classify into two classes "satisfied" and "dissatisfied". 
Then find out attributes which are the focused of "satisfied" and "dissatisfied" assessments, so that service 
providers can properly evaluate and improve the system [11]. Machine learning is an efficient way to extract 
knowledge from large amounts of data [12]. The problem in analyzing public service conversations is the large 
number of non-standard words that are difficult to understand by the system. The NLP approach is needed to 
improve the language of the conversation so that it is more easily understood by the system. Without NLP, 
machine learning cannot make meaningful progress [13]. 


2. RESEARCH METHOD 

The flow of research sentiment analysis of conversations in public services using the KNN and 
TF-IDF algorithm by using the NLP approach is explained in Figure |. The initial step of the study was the 
collection of recorded conversations of front desk services at one of the hospitals in Yogyakarta, Indonesia on 
July 13, 2020. Conversation recordings were converted into text using the "Kata Kita" application with the 
speech recognition system from Google speech-to-text [14]. The next step, text was stored in a MySQL 
database so that it is easily managed [15]. The workflow of the speech to text system in the "Kata Kita” 
application is explained in Figure 2. 






Change Speech to Text 
Data Labeling 


Data Training Data Testing 














KNN Calculation with TF-IDF 


Satisfaction Analysis 


Figure 1. Research stages 
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Figure 2. The process of changing the voice of conversation into text 


The next step, conversation text was then purchased manually by three respondents, consisting of 
customers, public relations and medical students. The dataset of 250 conversations was labeled in two classes, 
"satisfied" and "dissatisfied". The last step, dataset is subjected to a preprocessing process using the NLP 
approach. Preprocessing data is the process of cleaning and preparing data for analysis [16]. The purpose of 
preprocessing is to correct the language in the conversation, because in the conversation found many uses of 
non-standard words and local languages. In the case of text classification, many preprocessing techniques can 
be used [17]. The workflow of the preprocessing stage with the NLP approach is explained in Figure 3. 


Lowercase folding Word Normalizer 





Figure 3. The preprocessing stage uses the NLP approach 


NLP is a breakthrough in transcending language barriers [16]. NLP is a branch of Artificial Intelligence 
that focuses on processing natural language effectively and accurately as humans do [18]. The NLP features used in 
this study are as follows : 

— Lowercase folding 

Lowercase folding is changing all tokens to lowercase letters [19]. Conversation “Selamat pagi kak 
ada yang bisa dibantu” will be changed to “selamat pagi kak ada yang bisa dibantu”, alphabet “S” in the 
form of capital is changed into”s” in small form. 

— Word normalizer 

The word normalizer feature is used to correct words in sentences, so that a good and correct sentence 
is produced according to grammar rules [20]. Conversation “selamat pagi kak ada yang bisa dibantu” then 
after the Word Normalizer process becomes “salam kakak ada yang bisa dibantu”. Term “selamat pagi” 
changed into “salam” and term “kak” changed into “kakak”. 

— Stemming 

The stemming feature functions is to reduce words to basic words by eliminating the affixes that exist 
in those words [21]. For example, sentences “salam kakak ada yang bisa dibantu” changed into “salam kakak 
ada yang bisa bantu”, term “dibantu” changed to basic words “bantu”. 

— Stopword removal 

The stopword removal feature works to eliminate words that often appear in natural language but have 
very little meaning [22]. For example sentences “salam kakak ada yang bisa bantu” changed into “salam 
kakak bisa bantu ”, term “ada” and “yang” removed. 
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Term frequency is a method for finding the weight of a document by looking for the number of 
occurrences of terms in the document. The more often the term appears, it will affect the amount of weight and 
the suitability value of the document. Inverse Document Frequency is a method for calculating the distribution 
of terms in a document [23]. The TF-IDF method allows documents to be classified into two classes (positive 
and negative) [24]. TF-IDF calculation using (1). 


N 
Wey = tf 10g GH (1) 


Wy is the weight of the term (4) of document (dx). While t Tey is the number of occurrences of term (t,) in 


document (dx). N is the number of documents in the database and df, is the number of documents containing 
the term (£), there is at least one word, term (t). 


3. RESULTS AND ANALYSIS 
3.1. Service conversation text 

Conversation text used is obtained from the activity of changing conversations into text. This service 
conversation was taken in July 2020 at a hospital in Yogyakarta, Indonesia. Examples of conversations are 
shown in Table 1. Based on Table 1, Customer-1 was redeeming drugs at the hospital pharmacy and is satisfied 
with the service he received. Customer-2 was paying medical expenses at the cashier, but he had been 
transferred to another cashier for using Badan Penyelenggara Jaminan Sosial Ketenagakerjaan (BPJS) without 
saying an apology, so the customer felt dissatisfied. Customer-3 was talking to doctors with a mixture of 
Javanese words about the process of recovering hands, and feeling satisfied with the services provided. 
Customer-4 was dissatisfied when he was in a queue position, and he must leave the hospital temporarily, while 
the queue order will return from the beginning if the customer did not return when he was called. Customer-5 
was in a child's poly for immunizing his child and he was feeling satisfied because the doctor can calm the 
customer. Information from this conversation can be processed to be a future improvement. 


Table 1. Example conversation 
No Customer Conversation Label 

i| Customer-1 Assalamualaikum ada yang bisa saya bantu kak saya ingin menebus resep ini mbak silahkan Satisfied 
duduk dulu kak saya akan menyiapkannya ini obatnya kak yang tablet hijau diminum 45 menit 
sebelum makan tiga kali sehari yang kapsul biru dan putih masing masing satu setelah makan 
tiga kali sehari ada yang bisa dibantu lagi kak tidak terima kasih mbak 

2 Customer-2 Permisi bu saya ingin melakukan pembayaran atas nama windari ibu menggunakan bpjs Dissatisfied 
kesehatan atau reguler bu pakai bpjs bu untuk bpjs kesehatan bisa dilayani di kasir sebelah ya 
bu sini khusus yang reguler 

3 Customer-3 Gimana tangan saya dok tangan sampean sudah membaik pak kapan bisa lepas gips nya dok Satisfied 
gips dapat dilepas sekitar dua minggu lagi pak bapak dibanyakin makan ikan ikanan ya pak 
biar cepet pulih tangannya siap dok terima kasih 

4 Customer-4 Pak iya mbak antriannya kan masih lama kalo saya tingal dulu boleh tidak boleh mbak tapi Dissatisfied 
nanti kalo tiba urutan mbaknya gak ada nanti harus mengulang dari awal ya mbak kok gitu 
pak harusnya kalo kelewat ya disusulkan saja to pak maaf mbak tidak bisa 

5 Customer-5 Siang bunda benar adeknya bernama fahmi benar dok untuk imunisasi ya iya dok apakah nanti Satisfied 
badanya panas dok bunda gak perlu kawatir nanti panasnya cuman sebentar kok bisa 
dikompres biar cepet reda panasnya terima kasih dok 


3.2. Conversation analysis 

The data used 250 conversations, which were labeled by 3 correspondents and resulted in 125 
"satisfied" conversations and 125 "dissatisfied" conversations. The conversation is made into a dataset with a 
composition of 200 training data and 50 testing data, then preprocessing using the NLP approach. At the 
preprocessing stage the word normalizer, stemming and stopword removal features are applied for each 
conversation. 

Text normalization is a challenge to find the same words with different word variations [25]. Word 
normalizer can handle variations of writing words which have the same meaning so that they are counted as a 
single term in the calculation of the TF-IDF algorithm. Examples of variations in writing words which have 
the same meaning are shown in Table 2. In Table 2 shows variations of words which have the same meaning. 
Term “pak”, “bu”, “dok”, “kak”, “dik”, “mas” and “mbak” used to refer the other person, so that it can be 
replaced with term “sodara”. Term “selamat sore”, “selamat siang” and “assalamualaikum” used to give 
greetings so that it can be replaced with term “salam”. 

The next stage is the process of stemming, to eliminate the prefixes, inserts and word suffixes so that 
it becomes the basic form. Stemming is an important technique in NLP for efficient and effective information 
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retrieval [26]. An example of stemming application is shown in Table 3. In Table 3 terms “diperiksa”, 
“diperiksakan” dan “memeriksa” omitted prefixes, inserts and endings will be the same basic term is “periksa”. 
Terms “mengantar” and “diantar” after going through the stemming process, it becomes the same basic term 
is “antar”. Stemming makes a word into its basic form and becomes the same term. 


Table 2. Variations of words with the same meaning 
No Term on conversation Word normalizer results 


1 Pak saudara 
2 Bu saudara 
3 Dok saudara 
4 Kak saudara 
5 Dik saudara 
6 Mas saudara 
7 Mbak saudara 
8 Selamat Sore salam 

9 Selamat Siang salam 

10 Assalamualaikum salam 


Table 3. The results of applying stemming 
No Term on conversation Stemming results 


l Diperiksa periksa 
2 Diperiksakan periksa 
3 Memeriksa periksa 
4 Mengantar antar 
5 Diantar antar 
6 Diambil ambil 
7 Mengambil ambil 
8 Ditunggu tunggu 
9 Menunggu tunggu 
10 mendaftar daftar 


The next step is the stopword removal process to remove the stopword from the conversation. The 
stopword list which is used is made by itself, refers to the context of words which are often used in hospital 
services and numbers. Number have no effect on sentiment analysis, and removing them can reduce noise and 
increase efficiency [27]. An example of a stopword list used is shown in Table 4. Can be seen in Table 4, the 
term "yang" is one of the most frequently stoppedword lists, which is 204 times, and the term "di" which 
appears 101 times. Stopwords can be removed because they usually appear in large numbers and do not have 
meaning as a single term. The overall results of the preprocessing process can be seen in Table 5. Table 5 
shows conversations which have passed preprocessing using the NLP approach with the word normalize, 
stemming and stopword removal features. One indicator of satisfied customers is to say "terima kasih" at the 
end of the conversation which indicates that customers feel helped and are satisfied with the services provided. 

In conversation number 2, there is a mistake of the customer because of the wrong choice of the 
cashier, but customers disappointed cause is that no words of forgiveness make customers feel comfortable. 
Apology deals with the relationship between individuals involved in the wrong situation [28]. The choice and 
word usage in public services affect the level of customer satisfaction, although service personnel must be strict 
with the rules, with the right choice of words can maintain customer feelings. 


Table 4. Example stopword list 


No Stopword list Frequency of appearance 


1 yang 204 
2 di 101 
3 tapi 92 
4 dan 73 
5 ya 61 
6 juga 60 
7 jadi 36 
8 untuk 30 
9 dengan 30 
10 ke 23 
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Table 5. Preprocessing results 


Preprocessing results 


ne Conyereuen Word Normalizer Stemming Stopword Removal 
1 Assalamualaikum ada yang salam ada yang bisa saya bantu salam ada yang bisa saya salam ada bisa saya 


bisa saya bantu kak saya ingin 
menebus resep ini mbak 


objek saya ingin menebus resep 
ini objek silahkan duduk dulu 


bantu objek saya ingin tebus 
resep ini objek silah duduk 


bantu objek saya 
tebus resep objek 


silahkan duduk dulu kak saya objek saya akan dulu objek saya akan siap ini silah duduk objek 
akan menyiapkannya ini menyiapkannya ini obatnya obat objek yang tablet hijau saya siap obat objek 
obatnya kak yang tablet hijau objek yang tablet hijau minum 45 menit belum tablet hijau minum 


diminum 45 menit sebelum 
makan tiga kali sehari yang 
kapsul biru dan putih masing 
masing satu setelah makan tiga 
kali sehari ada yang bisa 
dibantu lagi kak tidak terima 
kasih mbak 


diminum 45 menit sebelum 
makan tiga kali sehari yang 
kapsul biru dan putih masing 
masing satu setelah makan tiga 
kali sehari ada yang bisa 
dibantu lagi objek tidak terima 
kasih objek 


makan tiga kali hari yang 
kapsul biru dan putih masing 
masing satu telah makan tiga 
kali hari ada yang bisa bantu 
lagi objek tidak terima kasih 
objek 


belum makan kapsul 
biru putih makan ada 
bisa bantu objek tidak 
terima kasih objek 


2 Permisi bu saya ingin salam objek saya ingin salam objek saya ingin laku salam objek saya laku 
melakukan pembayaran atas melakukan pembayaran atas bayar atas nama windari bayar nama windari 
nama windari ibu nama windari objek objek guna bpjs sehat atau objek guna bpjs sehat 
menggunakan bpjs kesehatan menggunakan bpjs kesehatan reguler objek pakai bpjs reguler objek pakai 
atau reguler bu pakai bpjs bu atau reguler objek pakai bpjs objek untuk bpjs sehat bisa bpjs objek bpjs sehat 
untuk bpjs kesehatan bisa objek untuk bpjs kesehatan bisa layan di kasir belah ya objek bisa layan kasir belah 
dilayani di kasir sebelah ya bu dilayani di kasir sebelah ya sini khusus yang reguler objek khusus reguler 
sini khusus yang reguler objek sini khusus yang reguler 

3 Gimana tangan saya dok bagaimana tangan saya objek bagaimana tangan saya tangan saya objek 
tangan sampean sudah tangan objek sudah membaik objek tangan objek sudah tangan objek sudah 
membaik pak kapan bisa lepas objek kapan bisa lepas gips nya baik objek kapan bisa lepas baik objek kapan bisa 
gips nya dok gips dapat dilepas objek gips dapat dilepas sekitar gips objek gips dapat lepas lepas gips objek gips 
sekitar dua minggu lagi pak dua minggu lagi objek objek sekitar dua minggu lagi objek dapat lepas sekitar 
bapak dibanyakin makan ikan dibanyakin makan ikan ikanan objek banyak makan ikan objek objek banyak 
ikanan ya pak biar cepet pulih ya objek biar cepet pulih ikan ya objek biar cepet pulih makan ikan ikan 
tangannya siap dok terima tangannya siap objek terima tangan siap objek terima objek cepet pulih 
kasih kasih kasih tangan siap objek 

terima kasih 

4 Pak iya mbak antriannya kan objek iya objek antriannya kan objek iya objek antri kan objek iya objek antri 
masih lama kalo saya tingal masih lama kalo saya tingal masih lama kalo saya tingal masih lama saya 
dulu boleh tidak boleh mbak dulu boleh tidak boleh objek dulu boleh tidak boleh objek tingal boleh tidak 
tapi nanti kalo tiba urutan tapi nanti kalo tiba urutan tapi nanti kalo tiba urut boleh objek tapi kalo 
mbaknya gak ada nanti harus objek tidak ada nanti harus objek tidak ada nanti harus tiba urut objek tidak 
mengulang dari awal ya mbak mengulang dari awal ya objek ulang dari awal ya objek kok ada harus ulang awal 
kok gitu pak harusnya kalo kok gitu objek harusnya kalo gitu objek harus kalo lewat objek gitu objek 
kelewat ya disusulkan saja to kelewat ya disusulkan saja to ya susul saja to objek maaf harus lewat susul 
pak maaf mbak tidak bisa objek maaf objek tidak bisa objek tidak bisa saja objek maaf objek 

tidak bisa 

5 Siang bunda benar adeknya siang objek benar adeknya siang objek benar adek nama siang objek benar 
bernama fahmi benar dok untuk bernama fahmi benar objek fahmi benar objek untuk adek nama fahmi 
imunisasi ya iya dok apakah untuk imunisasi ya iya objek imunisasi ya iya objek apa benar objek imunisasi 
nanti badanya panas dok bunda apakah nanti badanya panas nanti badan panas objek objek apa badan 
gak perlu kawatir nanti objek objek tidak perlu kawatir objek tidak perlu kawatir panas objek objek 


panasnya cuman sebentar kok 
bisa dikompres biar cepet reda 
panasnya terima kasih dok 


nanti panasnya cuman sebentar 
kok bisa dikompres biar cepet 
reda panasnya terima kasih 
objek 


3.3. KNN calculation using TF-IDF 
This study uses the KNN algorithm with k=3 and the term weighting uses TF-IDF. TF-IDF reflects 
the importance of a word in a text document [29]. The TF-IDF calculation is applied to 200 training data and 
50 testing data after preprocessing using the NLP approach. The next step is calculating the number of matches 
labeling results from the prediction of 50 testing data compared with manual labeling by correspondents using 
confusion matrix. The confusion matrix divides the results of the predictions into four categories, they are; 
1) true positive (TP) shows the amount of data with positive classes and true predictive results, 2) true negative 
(TN) shows the amount of data with negative classes and true predictive results, 3) false positive (FP) shows 
the number of data with positive classes and false, and 4) false negative prediction results (FN) shows the 
number of data with negative class and wrong prediction results. Prediction results are shown in Table 6. 

The results of the confusion matrix are performed calculations to get the value of accuracy, precision 
and recall. Accuracy is the proportion of the total number of correct predictions compared to the total number 
of samples, accuracy is calculated using (2) [30]. Precision is the proportion of positive samples which are 


nanti panas cuman sebentar 
kok bisa kompres biar cepet 
reda panas terima kasih 
objek 
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correctly classified against the positive total number of samples, precision is calculated using (3). A recall is a positive 
sample that is classified correctly to the total number of positive samples, recall is calculated using (4) [31]. 








TP+TN 

Accuracy = ——————_ (2) 
TP+FP+FN+TN 

Pe TP 

Precision = (3) 

FP+TP 
TP 
Recall = (4) 
TP+FN 


In (2-4) are applied to the test data testing results and the results are as shown in Table 7 

The measurement results in Table 7 show that labeling using KNN and TF-IDF with the NLP approach 
produces an accuracy 74.00%, precision 76.00% and recall 73.08%. Overall the test results showing positive 
results show machine learning has good performance. Previous research [9] using KNN and TF-IDF algorithms 
in classifying sentiments on Twitter social media obtained an accuracy of 67.2%, compared to this study 
obtained a higher accuracy of 74.00%. Higher accuracy values indicate that the use of NLP for preprocessing 
provides an increase in labeling accuracy. 


Table 6. Hasil Confusion matrix Table 7. Labeling test results 
Prediction results Number of documents Measurement Measurement results 
True Positif (TP) 19 Accuracy 74.00% 

True Negatif (TN) 18 Precision 76.00% 
False Positif (FP) 6 Recall 73.08% 
False Negatif (FN) 7 


3.4. Term occurrence frequency 

Terms which has a high frequency of occurrence in conversation can describe market acceptance of 
service quality. The terms often used in conversations are indicated by wordcoud created with the “Wordart” 
application in Figure 4 and Figure 5. In conversations with the label "satisfied", the term which has the highest 
frequency of occurrence is the term “akan” in 143 times, term “ingin” in 98 times, term “apakah” in 78 times, 
term “baik” in 54 times, term “terima” in 52 times and term “kasih” in 51 times. Term “apakah” and “ingin” 
refer to customer questions and desires, while terms “baik”, “terima” and “kasih” refers to customer satisfaction 
because questions or desires can be realized. 

In conversations with the label "satisfied", the term that has the highest frequency of occurrence is the 
term “tidak-bisa” in 138 times, term “lama” in 105 times, term “apa” in 75 times, term “BPJS” in 70 times and 
term “kira” in 55 times. Term “apa” and “tidak-bisa” shows customers do not get what is expected. Term 
“Jama” and “kira” shows that the customer is not comfortable waiting. Term “BPJS” shows the use of BPJS 
for payment of service fees. 


PuluuDahulu AAT > 
jenebus © Windar 


atus ` f Lima 
Pad Bâki (ae Masing seca 
Ale ah Mohon glunggu Terima SegeraMemakai Hubungi 
JA ho: dar Apa. 
Da aka Lima . Si He ans 
Pee. e Bervariasi 


jad ntar Bpjs njalani 
ey ee Terima m face rangai K [Pale 


@ Meat 





Makan 45 B Agar Tidak a Bagaimana o ‘ 
‘ Iya ibu 
TAN: ra | | idak-Bis ar 
| Y Untuk na Rib Masing” Unt Uu 1A Patah Kesehatan 
ve 
ang YIr ncn raf AM aRay 
Bisa In Hasil Kondisi e ne 
ioe Kali Keluarga a | n Qin anes 
In 2 Kartu any nbi uhar à her an 
Kasih" Kasih 
Figure 4. Word cloud appearance of words in a Figure 5. Word cloud the appearance of words in a 
satisfied conversation conversation is not satisfied 


TELKOMNIKA Telecommun Comput El Control, Vol. 19, No. 1, February 2021: 146 - 154 


TELKOMNIKA Telecommun Comput El Control O 153 


4. CONCLUSION 

Sentiment analysis on public service conversations can provide deeper information about customer 
ratings on services provided. Utilization of the speech recognition system technology can convert voice 
conversations into text, thus helping the process of analyzing customer satisfaction. Classification using the 
KNN and TF-IDF algorithm with optimization using the NLP approach at the preprocessing stage can result 
in better labeling of the conversation data. Testing with confusion matrix produces a value 74.00% of accuracy, 
76.00% of precision and 73.08% of recall. 

This research found that terms are dominant in conversation “satisfied” is term “akan”, “ingin”, 
“apakah” , “baik”, “terima” and “kasih” which shows customer satisfaction on the service received and 
customer desires that can be realized. Whereas the dominant terms in the conversation “dissatisfied” is term 
“tidak-bisa’’, “lama”, “apa”, “BPJS” and “kira” shows customer disappointment at the service received. This 
research is a series of studies on the product "Kata Kita" to be applied in public services that function to record 
conversations in the form of text, labeling auto learning based on machine learning and digging information. 
Information extracted from service conversations can be used as material for evaluating and developing service 
quality. Further research can be carried out on other features of "Kata Kita" products or with a variety of 
datasets from different public service sources. 
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